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STATEFUL FAIL-OVER OF SERVICE FIG. 1 is a block diagram illustrating a prior art system for 

MANAGERS providing a network service. A group of clients 101, 102, 

and 103 are connected by a network 110 to a group of servers 

CROSS REFERENCE TO RELATED 121, 122, 123, and 124. A network service appliance 130 is 

APPLICATIONS 5 physically located in the path between the clients and the 

. . servers. Network service appliance 130 provides a service 

v, f^T™ * rClated l ° U S - pate0t a PP hcaUon Ser ' by filtering packets, sending packets to specific destinations, 

No. 09/346,634 now U.S. Pat. No. 6,628,654 entitled DIS- or> ^ caseS( modifying the contents of packets. An 
PATCHING PACKETS FROM A FORWARDING AGENT example of such modification would be modifying the 

USING TAG SWITCHING; co-pending U.S. patent appli- packet header by changing the source or destination IP 

cation Ser. No. 09/347,124 entitled CASCADING MUL- address and the source or destination port number. 

TIPLE SERVICES ON A FORWARDING AGENT; Network service appliance 130 provides a network ser- 

co-pending U.S. patent application Ser. No. 09/347,111 vice such as load balancing, caching, or security services. In 

entitled LOAD BALANCING USING DISTRIBUTED providing security services, network service appliance 130 

FORWARDING AGENTS WITH APPLICATION BASED may function as a proxy, a firewall, or an intrusion detection 

FEEDBACK FOR DIFFERENT VIRTUAL MACHINES; device. For purposes of this specification, a network service 

U.S. patent application Ser. No. 09/347,428 now U.S. Pat. appliance that acts as a load balancer will be described in 

No. 6,606,316 entitled GATHERING NETWORK STATIS- detail. It should be noted that the architecture and methods 

TICS IN A DISTRIBUTED NETWORK SERVICE ENVI- described are equally applicable to a network service appli- 

RONMENT; co-pending U.S. patent application Ser. No. ance that is functioning as one of the other above described 

09/347,122 entitled HANDLING PACKET FRAGMENTS 20 devices 

IN A DISTRIBUTED NETWORK SERVICE ENVIRON- Network seiA , ice mhaaoo i 30 is physically located 

MENT; U.S. patent application Ser. No. 09/347,108 now between the group of servers and the clients that they serve. 

U.S. Pat. No. 6,549,516 entitled SENDING INSTRUC- ^ are disadV antages to this arrangement. First, it 

TIONS FROM A SERVICE MANAGER TO FORWARD- * difficult t0 add agonal network service appliances 

ING AGENTS ON A NEED TO KNOW BASIS; U.S. patent whcn ^ first nctwork XTvicc appIiance bcco ™ s over . 

application Ser. No. 09/347,126 now U.S. Pat. No. 6,033, loaded because mc physical connections of the network 

560 entitled DISTRIBUTION OF NETWORK SERVICES must ^ rerouted Likewisef it is difficu i t to replace the 

AMONG MULTIPLE SERVICE MANAGERS WITHOUT network ^wice appliance with a back up network service 

CLIENT INVOLVEMENT; co-pending U.S. patent appli- appliance when it fails. Since all packets pass through the 

cation Ser. No. 09/347,034 entitled INTEGRATING SER- network liance oc the to the w tDe 

VICE MANAGERS INTO A ROUTING INFRASTRUC- failure of the QetW ork service appliance may prevent any 

TURE USING FORWARDING AGENTS; U.S. patent packets from reaching the ^ ^ any packets from 

application Ser. No. 09/347,048 now U.S. Pat. No. 6,606, being sent by the servers. Such a single point of failure is 

315 entitled SYNCHRONIZING SERVICE INSTRUC- UQdes irable. Furthermore, as networks and internetworks 

TIONS AMONG FORWARDING AGENTS USING A have become increasingly complex, multiple services may 

SERVICE MANAGER; co-pending U.S. patent application be requ i rcd f or a single network and inserting a large number 

No. 09/347,125 entided BACKUP SERVICE MANAGERS ofnct work service appliances into a network in places where 

FOR PROVIDING RELIABLE NETWORK SERVICES IN ±cy can iai t Relevant packet flows may be imprac- 

A DISTRIBUTED ENVIRONMENT; co-pending U.S. tical 

patent application Ser. No. 09/347,109 entitled NETWORK 40 ^ ^ bc rcfcrrcd , fe and me 

ADDRESS TRANSLATION USING A FORWARDING of ^ may £ bc rcfcrrcd tQ „° a ^ of b^ft^ 

AGENT; and co-pending U.S. patent application Ser. No. f h ^mmon IP address, that IP address 

^^V^r^ 1 ^^^^^ « ^ be refe ' red 10 35 a - rtual IP address (%PA) o7 a cluste" 

CONNEC^ON USING A FORWARDING AGENT, all ^ ^ u shou]d be noted ^ ^ ^ ^ aQd 

filed on Jul. 2, 1999 and incorporated herein by reference for server are used herein m a genefal MQac tQ refer t0 devices 

a purposes. tbaI g enera Uy re q Ues t information or services (clients) and 

FIELD OF THE INVENTION devices that generally provide services or information 

(servers). In each example given it should be noted that the 

The present invention relates generally to providing net- 50 roles of client and server may be reversed if desired for a 

work services such as load balancing, packet filtering or particular application. 

Network Address Translation (NAT). More specifically, net- A system that addresses the scalability issues that are 

work services are provided using service managers and f aced by network service appliances (load balancers, 

forwarding agents that are integrated into a routing infra- firewalls, etc.) is needed. It would bc useful to distribute 

structure. 5S functions that are traditionally performed by a single net- 

BACKGROUND OF THE INVENTION W ° rk elemcnt and 50 tDat 25 mucD flu,ction as possible can 

be performed by multiple network elements. A method of 

As the IP protocol has continued to be in widespread use, coordinating work between the distributed functions with a 

a plethora of network service appliances have evolved for minimum of overhead is needed. 

the purpose of providing certain network services not 60 Although network service appliances have facilitated the 

included in the protocol and therefore not provided by development of scalable server architectures, the problem of 

standard IP routers. Such services include NAT, statistics scaling network service appliances themselves and distrib- 

gathering, load balancing, proxying, intrusion detection, and uting their functionality across multiple platforms has been 

numerous other security services. In general, such service largely ignored. Network service appliances traditionally 

appliances must be inserted in a network at a physical 65 have been implemented on a single platform that must be 

location where the appliance will intercept all flows of physically located at a specific point in the network for its 

interest for the purpose of making its service available. service to be provided. 
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For example, clustering of servers has been practiced in readable medium such as a computer readable storage 

this manner. Clustering has achieved scalability for servers. medium or a computer network wherein program instruc- 

Traditional multiprocessor systems have relatively low seal- tions arc sent over opticaj or electronic communication 

ability limits due to contention for shared memory and I/O. links. Several inventive embodiments of the pre sent inven- 

Clustcrcd machines, on the other hand, can scale farther in 5 non are described below. 

that the workload for any particular user is bound to a la one embodiment, a fault tolerant method of providing 

particular machine and far less sharing is needed. Clustering a network service includes receiving a packet corresponding 

has also facilitated non-disruptive growth. When workloads to a flow from a forwarding agent at a primary service 

grow beyond the capacity of a single machine, the traditional manager and determining at the primary service manager 

approach is to replace it with a larger machine or, if possible, to ^mictions &>r handling packets corresponding to the flow, 

add additional processors within the machine. In either case, V* instnicll0ns are *f nt <° *e forwarding agent and the 

this requires downtime for the entire machine. With instructions are stored at the primary service manager A 

clustering, machines can be added to the cluster without Ration packet is sent to a backup service manager. The 

disrupting work that is executing on the other machines. ^phcation packet includes the instructor* for handling 

When the new machine comes online, new work can start to 15 P acketS «>™sP™ding to the Bow. 

migrate to that machine, thus reducing the load on the In another embodiment, a primary service manager for 

pre-existing machines. providing a network service m a fault tolerant manner 

~. . . jjijl, • includes a processor configured to determine instructions for 

Clustering has also provided load balancing among serv- ^ * corresponding to a flow. A forwarding 

ers. Spreading users across multiple ^dependent systems fc am ^ fmd * seQd ^ ^ructions & 

can result in wasted capacity on some systems while others 20 ^ , . , n f * _ . A m „ 

i j j r. i • i j i_ i - -.l- handling packets to a forwarding agent. A memory is con- 

are over oaded. By employing load balancing within a fi ^ store me instructions for handling packets corre- 

clustcr of sys ems the users are spread to available systems ^ tQ ^ flow Abacfcu ^ 

based on the load on each system Clustering also has been c £j fi ^ to ^ a lic ^ n ket to a £ acku savia 

used to enable systems to be continuously available. Inch- ° , ... t * i i j • . 

... . . , , . t« manager wherein the replication packet includes instructions 

vidual application instances or machines can fail (or be 25 r , & . t r J. „ a 

t . m p . t v ... t . „. . v for handling packets corresponding to the flow, 

taken down for maintenance) without shutting down service , , , , 

_ j ,„.„,„ „ p ■ | _j „.„,„„ „ m „„, A In another embodiment, a backup service manager for 

to end-users. Users on the failed system reconnect and ... , . . , , , 

should not be aware that they are using an alternate image. P r ° vidin g a nctwork «™ce in a fault tolerant manner 

Users on the other systems are completely unaffected except eludes a primary service manager interface configured to 

for the additional load caused by services provided to some 30 ««ive the instructions for handling packets corresponding 

portion of the users that were formerly on the failed system. *° a now. Amemory is configured to store the instructions for 

„ „ „ „ handling packets corresponding to the flow. 

In order to take full advantage of these features, the T t . ... . c Ui . t ,. t .. t , 

... . . 6 . . . ..... In another embodunent, a fault tolerant distributed system 

network access must likewise be scalable and highly avail- r ... , i • • i j c _j* 

, i . . . /t . , . . for providmg a network service includes a forwarding agent 

able. Network service appliances (load-balancing appliances % , , , . . « 7 

. . . . r \ , . ■ it • i_ 4 35 configured to send a packet corresponding to a now to a 

being one such example) must be able to function without * ■ 

~. ,. .... , ._, . primary service manager. A primary service manager is 

introducing their own scaling limitations that would restrict r « „. ... - - . c ■ . , . 

. . 7 . r , " & . . . r . . . configured to determine instructions for handling packets 

the throughput of the cluster. A new method of providing corre * di to the flow md tQ ^ me mstm ttions for 

l^eb™™™ 1 * 8, adlstnbutcdarcmtectureisnccdcdt0 handling packets to the forwarding agent. The primary 

40 service manager stores the instructions for handling packets 

In addition to being highly available, it would be useful if corresponding to the flow and sends a replication packet to 

network services could be provided in a manner such that a backup service manager. The replication packet includes 

there is a smooth transition between a primary service ^ instructions f or handling packets corresponding to the 

manager and a backup service manager when the primary flow. A backup service manager is configured to receive the 

service manager tails. An efficient and reliable method of 45 instructions for handling packets corresponding to the flow 

transferring state information between the primary service and t0 store the instructions for handling packets corre- 

manager and the backup service manager is needed. sponding to the flow. 

SUMMARY OF THE INVENTION These and other features and advantages of the present 

invention will be presented in more detail in the following 

A system that includes a primary service manager and a so specification of the invention and the accompanying figures 

backup service manager is disclosed. The primary service which illustrate by way of example the principles of the 

manager determines how a network service is provided and invention. 

sends instructions to the forwarding agents that cause the nFsrR.PnnNT r>F thf drawings 

forwarding agents to take appropriate actions. The primary BRIEF DESCRIPTI0N OF THE DRAWINGS 

service manager keeps track of flows that are being serviced 55 The present invention will be readily understood by the 

and maintains instructions for the flows according to the following detailed description in conjunction with the 

traffic that the service manager monitors for the flows. In accompanying drawings, wherein like reference numerals 

addition to sending instructions to the forwarding agents, the designate like structural elements, and in which: 

primary service manager also copies the instructions to the FIG. 1 is a block diagram illustrating a prior art system for 

backup service manager, which maintains the instructions in g 0 providing a network service. 

parallel with the primary service manager. When the primary FIG. 2A is a block diagram of a network architecture that 

service manager fails, the backup service manager may provides network services without requiring a network ser- 

begin servicing the flows formerly serviced by the primary vice appliance to be physically placed at a node through 

service manager. which all incoming and outgoing packets processed by a 

It should be appreciated that the present invention can be 65 group of servers must pass, 

implemented in numerous ways, including as a process, an FIG. 2B is a block diagram illustrating an architecture for 

apparatus, a system, a device, a method, or a computer a forwarding agent. 
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FIG. 2C is a block diagram illustrating an architecture for DETAILED DESCRIPTION 

a service manager. A detailed description of a preferred embodiment of the 

FIG. 3A is a diagram illustrating how a service manager invenlioo is provided below. While the invention is 

and a forwarding agent cooperate to establish a connection described in conjunction with that preferred embodiment, it 

from a client to a selected real machine. 5 should be unde rstood that the invention is not limited to any 

FIG . 3B is a diagram illustrating how a forwarding agent one embodiment. On the contrary, the scope of the invention 

routes a SYN ACK returned from a host back to a client. is limited only by the appended claims and the invention 

FIG. 3C is a diagram illustrating how a subsequent data encompasses numerous alternatives, modifications and 

packet from client 304 is routed by forwarding agent 302 to equivalents. For the purpose of example, numerous specific 

host 306. 10 details are set forth in the following description in order to 

FIG. 4 is a diagram illustrating a network that includes provide a thorough understanding of the present invention, 

two forwarding agents and two service managers. The present invention may be practiced according to the 

FIG. 5 is a diagram illustrating how a service manager claims without some or all of these specific details. For the 

provides instructions to two separate forwarding agents for purpose of clarity, details relating to technical material that 

handling a connection. is known m the technical fields related to the invention has 

FIG. 6 is a diagram illustrating a fixed affinity. not described in detail in order not to unnecessarily 

FIG. 7 is a diagram illustrating a wildcard affinity. obscure me P rcseDt ^ntion in such detail. 

FIG. 8A is a diagram illustrating a service message FIG. 2A is a block diagram of a network architecture that 

neader 2Q provides network services without requiring a network ser- 

FIG: 8B is a diagram illustrating a segment header. ™ e , a PP! iance 10 te P^fSf P laced u « a node 

. , .,, which all incoming and outgoing packets processed by a 

FIG. 8C is a diagram illustrating a security message g^f^ers must pass. Several clients 201, 202, and 203 

^^^P™' . ,. .„ . „. arc connected to a network 210. Network 210 is connected 

FIG. 9A is a diagram illustrating an affinity update wild- to a group of mm m (hat indudes ^ 222> and 

card message. 223. There is no point through which all traffic between 

FIG. 9B illustrates a fixed affinity update message that is devices connected to network 210 and the group of servers 

sent by a service manager to a forwarding agent to add a 22 q must pass. Instead, some traffic from network 210 that 

fixed affinity to the receiver's affinity cache or delete a fixed [g bound for the group of servers passes through a forward- 

affinity that is stored in the receiver's affinity cache. ^ agenl 2 31 and some traffic between network 210 and 

FIG. 9C is a diagram illustrating an affinity update-deny group of servers 220 passes though a forwarding agent 232. 

message. j D tne exam pi e shown, forwarding agent 231 is connected 

FIG. 9D is a diagram illustrating an interest match mes- to server 221 and server 222 and forwarding agent 232 is 

sage for either a wildcard affinity or a fixed affinity. connected to server 222 and server 223. Thus, server 222 

FIG. 9E is a diagram illustrating an IP packet only 3S may communicate with network 210 through either of the 

message. forwarding agents, server 221 communicates with network 

FIG. 10A is a diagram illustrating an affinity identifier 210 exclusively through forwarding agent 231, and server 

segment. 223 communicates with network 210 exclusively through 

FIG. 10B is a diagram illustrating an affinity service forwarding agent 232. This arrangement may be generalized 

precedence segment. 40 to include an arbitrary number of servers connected to an 

FIG. 10C is a diagram illustrating a service manager arbitrary number of forwarding agents with individual serv- 

interest data segment. ers connected to arbitrary subsets of the forwarding agents. 

FIG. 10D is a diagram illustrating a forwarding agent A service manager 241 and a second service manager 242 

interest data segment. also communicate with the forwarding agents. The service 

FIG. 10E is a diagram illustrating an identity information 45 managers provide the decision making capability that is 

segment that is used to identify the sender of a service required to provide a network service such as load balanc- 

message ^S- service managers send specific instructions to each 

FIG. 10F is a diagram illustrating a NAT (Network of forwarding agents detailing how certain flows of 

Address Translation) action segment. ? &c ^ are '° te P««?8ed- fuch packet processing may 

FIG. 10G is a diagram illustrating a sequence number 50 mchioc pimply pouting the packet, gathering statistics about 

..... f & ™i the packet, sending the packet to a service manager, sending 

adjust action segment. *\ 7. f K . . , , t & . 

V™ . „ • , . a notification that the packet has been seen to a service 

FIG. 10H is a diagram illustrating an advertise action manager> modifying the or using a special melho d 

segment. such flS ^nu^g or tag switching to send the packet to a 

FIG. 101 is a diagram tllustrating an interest criteria 55 destination ot her than the destination specified by the des- 

action - tination IP address included in the packet header. It should 

FIG. 10J is a diagram illustrating an action list segment. ^ De noted that forwarding agents in other embodiments 

FIG. 11 is a block diagram illustrating a distributed also modify other aspects of packets, including packet 

network service architecture including service managers and source and destination addresses and port numbers and, in 

forwarding agents. ^ some instances, packet data. 

FIG. 12 is a flow chart illustrating a process executed by The service managers communicate with the forwarding 

a service manager for managing fixed affinities. agents to give the agents instructions relating to how to 

FIG. 13 is a flowchart illustrating the process for sending handle packets for various Sows that are routed through the 

a replication packet to the backup service manager. forwarding agents. It is useful at this point to review certain 

FIG. 14 is a flowchart illustrating a process implemented 65 terminology used herein relating to connections and flows. 

on the backup service manager upon the receipt of a repli- As used in this specification, a connection consists of a set 

cation packet. of flows. A flow is a set of related packets sent between two 
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end stations. A flow may be identified with layer 3 and layer 
4 parameters, depending on the protocol being used. For 
example, for TCP and UDP, a flow is identified by five 
parameters: the source and destination IP addresses and port 
numbers and the protocol. For ICMP, flows are defined by 
three parameters: the source and destination IP addresses 
and the protocol. 

TCP connections will be described in detail in this speci- 
fication. It should be appreciated that the techniques dis- 
closed apply to other types of connections as well. TCP 
connections are defined by a 5-tuple that includes the source 
and destination IP addresses, the source and destination port 
numbers, and an identification of the protocol that applies to 
the packet. The source and destination IP addresses and ports 
for packets going in one direction between the devices are 
reversed for packets going in the opposite direction. That is, 
when the direction that a packet is travelling is reversed, the 
source becomes the destination and the destination becomes 
the source. Packets flowing in one direction of a connection 
are in the same flow, 

A connection transfers data between applications on two 
machines having IP addresses and the applications corre- 
spond to port numbers. If the protocol is set by convention 
to be a certain protocol such as TCP, then a protocol 
identifier may not be required. The 4 remaining numbers, the 
source and destination IP addresses, and the source and 
destination port numbers, are sometimes referred to as a 
quad. In this specification, the 5-tuple that includes the 
source and destination IP addresses, the source and desti- 
nation port numbers and a protocol identification will be 
referred to as an affinity key. Each unique affinity key thus 
defines a flow in one direction of a connection. If the source 
and destination IP addresses and port numbers are reversed 
for a single affinity key, then it becomes an affinity key that 
corresponds to a flow in the opposite direction for the same 
connection. In general, a flow may be identified by a source 
IP address and destination IP address, by a source IP address, 
destination IP address and protocol, by a quad, by an affinity 
key 5-tuple, by only a source and destination IP address or 
by other information available in a packet header. The term, 
"flow identifier" is intended to refer to any such method of 
identifying a flow. 

Affinity keys are used by the service managers to identify 
flows passing through forwarding agents which are to be 
handled by the forwarding agents in a certain manner. 
Forwarding agents can accomplish their required tasks with 
only limited processing capability. Forwarding agents need 
not determine how to handle certain flows or make decisions 
such as load balancing or security decisions relating to the 
flows. The service manager performs those functions and 
forwards specific instructions to forwarding agents detailing 
exactly what actions are to be taken for each flow. Instruc- 
tions for how to handle packets are specified for each flow 
by the service managers using an affinity key. A specific 
affinity key that is sent to a forwarding agent together with 
instructions detailing how packets for flows specified by the 
affinity key are to be handled is referred to as a fixed affinity. 

In addition to specifying instructions for each flow, ser- 
vice managers must also obtain information about each new 
flow from the forwarding agents. For example, when a 
service manager provides load balancing through a set of 
forwarding agents, the service manager uses fixed affinities 
to provide specific instructions to the forwarding agents 
detailing where packets for each load balanced flow are to be 
forwarded. In addition to providing those specific 
instructions, the service manager also provides general 
instructions to each forwarding agent that specify which new 
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flows the service manager is interested in seeing. These 
general instructions are provided using wildcard affinities. 
Wildcard affinities, which are described in detail below, 
specify sets of flows that are of interest to a service manager. 

5 In one embodiment, this is done by specifying subnet masks 
that determine sets of source and destination IP addresses 
that will be forwarded to a service manager. In addition, 
ports or sets of ports and protocol may be specified in 
wildcard affinity as well. As is described further below, the 

10 use of wildcard affinities enables separate service managers 
to be configured to provide services for different sets of 
flows. Each service manager specifies the flows of interest to 
it and other service managers handle other flows. In this 
manner, service managers can be configured in parallel to 

15 share load. 

Thus, service managers use wildcard affinities to specify 
flows for which they may be providing service and forward- 
ing agents transfer packets for new flows to the appropriate 
service manager. Once a service manager determines how a 

20 certain flow is to be handled, the service manager sends a 
fixed affinity to each forwarding agent. The fixed affinity 
overrides the wildcard affinity stored in the forwarding agent 
that instructs the forwarding agent to forward packets to the 
service manager with specific instructions for the specific 

25 flow specified by an affinity key in the fixed affinity. 

In the case of load balancing, service managers send 
wildcard affinities to forwarding agents. The wildcard affini- 
ties specify destination IP addresses that correspond to 
virtual IP addresses of server clusters that are to be load 

30 balanced by the service manager. The forwarding agents 
then forward new packets sent to those virtual IP addresses 
to the appropriate service manager. The service manager 
selects a server from the server cluster and then the service 
manager sends a fixed affinity to each forwarding agent that 

35 instructs the forwarding agent to forward packets for that 
specific flow to the selected server in the cluster. Forwarding 
agents may also forward packets for purposes other than 
load balancing. Packets may be forwarded to real IP 
addresses as well as virtual IP addresses. 

40 

Id one embodiment, each forwarding agent is imple- 
mented on a router. In other embodiments, forwarding 
agents may be implemented on switches or other network 
devices and may be implemented on a coprocessor in a 

4S device that also performs another network function. When 
implemented on a router, the power of this architecture 
becomes clear. By infusing each router with a limited 
functionality provided by the forwarding agent, the service 
managers are able to provide network services without 

50 physically being inserted at the various points in the network 
where those services must be provided. The physical pres- 
ence of each of the routers at those points is sufficient to 
enable network services to be provided. This contradicts the 
conventional wisdom regarding the restriction that all traffic 

S5 inbound for a server cluster must pass through a single 
load-balancing engine. The combination of fast forwarding 
agents (be they 'routers' or IP-aware 'switches*) and service 
managers (to provide synchronization and control) elimi- 
nates the scalability limitations of the past. 

50 This specification will refer in detail to forwarding agents 
implemented on routers for the purpose of example. It 
should be remembered that forwarding agents may also be 
implemented on other devices and that the same or similar 
advantages may be realized. 

65 The service managers send wildcard affinities to each of 
the forwarding agents that direct the forwarding agents to 
process packets that match the wildcard affinities in a certain 
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manner. For example, a service manager may request to be processor 272 and a memory 274. Memory 274 may include 

notified when certain packets are received by the routers that RAM, ROM, nonvolatile memory such as an EEPROM or 

include the forwarding agents. When a packet that matches a disk drive. Service manager 270 also includes a user 

such an instruction is received, the forwarding agent notifies interface 276 for the purpose of allowing a user to configure 

the service manager and the service manager determines 5 th c service manager or monitor the operation of the service 

what to do with that packet and future packets for the flow manager. 

based on the network service being provided. Instructions 0 . . „ . , , , 

are then sent from the service manager to the forwarding . Serv.ce manager 270 also optionally mcludes a network 

agent at the router that allow the router to process the mterface 278 ". Netw ° rk mterfa <* 278 aUows tbe M ™ 

packets in accordance with the decisions made by the manager to directly forward packets into the network for 

service manager 10 wn ' cn k providing a service. If no network interface is 

In addition to specifying that a service manager is to be provided, then the service manager can still forward packets 

notified upon receipt of a certain type of packet, wildcard h V sending them to a forwarding agent, 

affinities may also specify other actions to be taken. For A forwarding agent interface 280 is included on the 

example, a wildcard may specify an IP address to which service manager for the purpose of allowing the service 

packets are to be forwarded without notification to the 15 manager to send packets and affinities to forwarding agents, 

service manager. Packets may also be copied to a service Forwarding agent interface 280 may include more than one 

manager or other device and packets may also be denied or interface. For example, in one embodiment, a separate 

dropped. interface is used for multicasting wildcard affinities to all 

It should be noted that the service managers also may be forwarding agents and a separate interface is used for the 
connected to one or more of the servers and may in some 20 purpose of unicasting fixed affinities to individual forward- 
cases forward packets received from forwarding agents or ing agents and forwarding packets to individual forwarding 
received from the network directly to certain servers. agents. 

However, it is significant that the service managers need not Service manager 270 may also include a service manager 
be connected to servers for which they are managing packet ^ interface 282 used to communicate with other service man- 
traflfic. The service manager may accomplish all packet agers. The service manager may communicate with other 
routing through forwarding agents by sending instructions to service managers for the purpose of providing a fail over 
forwarding agents. It should also be noted that the service scheme of backup service managers. Operational status of 
managers may also be connected to each other for the service managers may be communicated on the service 
purpose of coordinating their instructions or providing ^ manager interface and a master service manager may send 
backup services. configuration information about flows being supported 
FIG. 2B is a block diagram illustrating an architecture for through backup service managers so that the backup service 
a forwarding agent. Forwarding agent 250 includes a main managers can function in place of the master service man- 
processor 252 and a memory 254. Memory 254 may include ager should it fail. 

RAM, ROM, nonvolatile memory such as an EPROM, or a 35 A service manager may be implemented on a standard 
disk drive. Forwarding agent 250 also includes a user microcomputer or minicomputer. In one embodiment a 
interface 256 that allows a user to configure the forwarding service manager is implemented on a UNIX workstation. A 
agent or monitor the operation of the forwarding agent. Service manager may also be implemented on other plat- 
Forwarding agent 250 also includes a service manager forms including Windows, an embedded system or as a 
interface 258 that allows packets to be sent to and received ^ system on a chip architecture. A service manager also may 
from a service manager. In addition, the service manager be implemented on a router. 

interface allows service managers to send fixed and wildcard One network service that can be readily provided using 

affinities to the forwarding agent. In one embodiment, a mc architecture described in FIG. 2A is load balancing 

separate interface is used for the purpose of sending wild- connections among a set of real machines that are used to 

card affinities to forwarding agents using multicast. In other 45 service connections made to a virtual machine. The real 

embodiments, a single interface may be provided between machines may also be referred to as hosts and the virtual 

the service manger and the forwarding agent. The forward- machine may also be referred to as a cluster of hosts. The 

ing agent also includes a network interface 260 that is used following figures describe how a service manager directs 

to send and receive packets to and from other devices on thc forwarding agents to intercept packets for new connections 

network. 50 and send them to the service manager. The service manager 

It should be noted that thc network interface and the then selects a real machine to handle each connection, and 

service manager interface may be the same interface in directs one or more forwarding agents to forward packets to 

certain embodiments. In such embodiments, all communi- thc selected real machine. Forwarding agents may forward 

cation between the forwarding agent and the service man- packets using NAT or may use another method of sending 

ager is carried on the same network as packets processed by 55 packets to the selected real machine, 

the forwarding agent. FIG. 3A is a diagram illustrating how a service manager 

A forwarding agent may be implemented on various and a forwarding agent cooperate to establish a connection 

network devices. A forwarding agent may be implemented from a client to a selected real machine. A service manager 

on a network device dedicated to acting as a forwarding 300 broadcasts or multicasts a wildcard affinity to all for- 

agent but the true power of the system is realized when 60 warding agents that are listening for wildcard affinities sent 

forwarding agents are implemented on network devices that by service manager 300. In some embodiments, wildcard 

already arc included in a network for some other purpose. affinities may be broadcast. A forwarding agent 302 receives 

Forwarding agents may be implemented on routers that the wildcard affinity. In one embodiment, all forwarding 

already exist at strategic points in a network for intercepting agents and service managers register to a common multicast 

packets and providing a service using a forwarding agent. 65 group so that neither service managers nor forwarding 

FIG. 2C is a block diagram illustrating an architecture for agents need to have any preknowledge of the existence of 

a service manager. Service manager 270 includes a main each other. Thus, a service manager registers its interests 
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with the forwarding agents by multicasting wildcard affini- travels through forwarding agent 302. Because of the wild- 
ties to the multicast group. Each wildcard affinity provides card affinity that matches the source IP address of host 306, 
a filter which recognizes general classes of packets that are forwarding agent 302 encapsulates the SYN ACK packet 
of interest. and sends it to service manager 300. Service manager 300 
As an example, client 304 may wish to establish a TCP 5 (ben ide nttftt the SYN ACK as the SYN ACK correspond- 

connection with a virtual machine having a virtual IP "* 10 th ! T^™^? ^ ^^TT^^ 

, , r u . . . B nnt , t . t nt . t^r Z.r and sends the SYN ACK together with a fixed affinity to 

address. It should be noted that other types of connections forwardi t 302 ^ | xed affimt includ( f „ 

may abobc "tablrtetL To establish the TCP connection, ^ ±t f^as the forwarding agent to replace the source 

client 304 sends a SYN packet with a destination address , p address of host M ^ tbe s vu ?u a i IP of virtual 

corresponding to the virtual IP address. The SYN packet is 10 machine 3W forwardi tfac syN ACK kct Qn tQ 

received by forwarding agent 302. Forwarding agent 302 client 304 

determines that the destination address of the SYN packet FIGS 3A and 3B show how a forwarding agcnt 

matches the wildcard affinity broadcast by service manager intercepts a SYN packet from a client and translates the 

300. The action included in the broadcast wildcard affinity destination IP address from the destination IP address of a 

specifies that all packets matching the wildcard affinity are 15 virtual machine to the destination IP address of a specific 

to be forwarded to the service manager. Therefore, forward- host. The specific host is determined by the service manager 

ing agent 302 forwards the SYN packet to service manager using a load balancing algorithm. The forwarding agent does 

300. not include logic that performs load balancing to determine 

Service manager 300 receives the SYN packet from the ^ host - ^ forwarding agent only needs to check 

forwarding agent. It should be noted that, in one 20 whether the incoming SYN packet matches a fixed affinity 

embodiment, forwarding agent 302 encapsulates the SYN ° r a wildcard affinity broadcast to the forwarding agent by 

packet in a special system packet when the SYN packet is me service manager. 

sent to the service manager. Service manager 300 receives ' Ilie SYN P acket is forwarded to the service manager and 
the SYN packet and processes the packet according to the service manager returns the SYN packet to the forward- 
whatever service or services are being provided by the 25 m S a S ent ? lth * ^ ^ ^"f? 

service manacer In the examole shown service manager whlch specifies how the forwarding agent is to handle the 

service manager, in me example snown, service manager syN gt When a syN ACK ^ returned b ±e host the 

300 is Providing load balancing between a first host 306 and forwa J di t ^ finds a ^ J t matcr /and 

a second host 308. Together, host 306 and host 308 comprise forwards ^ SYN ACK kcl t0 thc m ^ 

a virtual machme that services the virtual IP address that is manager retums tbe SYN ACK packet to the for- 

thedestinationoftheSYNpacketsentbychent304.Service warding agent ^ ^ a fij£ed affinity that 

manager 300 determines the host that is to receive the SYN instructs the forwarding agent how to handle packets in the 

packet and that is to handle the connection initiated by the flow back from tne host tQe client 

SYN packet. This information h ; included in a fixed affinity ^ fifst ^ ^ from ^ ef 

s^nt blk Tf rwrrdrrenT^r 35 a ° key that corres P° nds to * e flow * e cUeDt 

sen ac orwar ing agent . to ^ host and the second fixed afiBnity sent form the service 

The fixed affinity sent to the forwarding agent 302 may manager to the forwarding agent contains an affinity key that 

include an action that directs the forwarding agent to dis- corresponds to the flow from the host back to the client, 

patch the SYN packet directly to host 306. Tbe actioo Future packets in either flow sent from the client or the' host 

, included in the fixed affinity may also direct the forwarding 4Q matc h the affinity key in one of the fixed affinities and are 

agent to translate the destination address of the packet to the handled by the forwarding agent according to the action 

IP address of host 306 and the packet may be routed to host contained in the fixed affinity. It is no longer necessary to 

306 via one or more hops. In addition, as described below, forward such packets to the service manager. In some 

tag switching may also be used to send the packet to the host applications, the forwarding agent may continue to forward 

that is selected by the service manager using its load 4S data aoout the packets to the service manager so that the 

balancing algorithm. service manager can monitor connections or maintain sta- 

Thus, the SYN packet is directed to the host selected by tistics about network traffic, 

service manager 300 without service manager 300 being FIG. 3C is a diagram illustrating bow a subsequent data 

inserted into the path of tbe packet between the hosts which packet from client 304 is routed by forwarding agent 302 to 

comprise virtual machine 310 and client 304. The service 50 host 306. Client 304 sends a data packet to forwarding agent 

manager broadcasts a wildcard affinity to all forwarding 302. Forwarding agent 302 has stored the fixed affinity 

agents potentially in that path and the forwarding agents corresponding to the flow from the client to the host in a 

forward SYN packets to the service manager whenever a fixed afiinity database 303. Forwarding agent 302 notes the 

client establishes a new connection. The service manager match of the 5-hiple of the data packet with an affinity key 

then retums the SYN packet with a fixed affinity that directs 55 in the fixed affinity database and then forwards the data 

the forwarding agent how to forward that SYN packet as packet according to the action defined in that fixed affinity, 

well as future packets sent in the flow from the client to the In this example, the action defined is to translate the desti- 

virtual machine. Tbe forwarding agent then sends the SYN nation IP address of the client from the virtual IP address of 

, packet on to the selected host using network address trans- virtual machine 310 to the IP address of host 306. In addition 

lation (NAT), tag switching, or some other method. 60 to forwarding the data packet, the affinity found by the 

FIG. 3B is a diagram illustrating how a forwarding agent forwarding agent also includes an action that requires the 
routes a SYN ACK returned from a host back to a client. A forwarding agent to send an affinity packet to service man- 
service manager 300 broadcasts a wildcard affinity to a ager 300 that includes data about the packet for the purpose 
forwarding agent 302. The wildcard affinity matches packets of service manager 300 gathering statistics about network 
with a source IP address matching either host 306 or host 65 traffic. 

308 which implement virtual machine 300. When host 306 The examples shown in FIG. 3A through FIG. 3C illus- 

sends a SYN ACK packet back to client 304, the SYN ACK trate how the first packet sent in both flows of a new 
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connection are forwarded to the service manager by the 
forwarding agent. The service manager then directs the 
forwarding agent to handle the packets in a certain manner 
by sending fixed affinities to the forwarding agent for each 
flow and specifying actions to be performed on the packets. 
In the example shown, the action involves translating the 
destination IP address from the client to a specific host IP 
address and translating the source IP address in packets form 
the host to a virtual IP address. Other actions may be defined 
by fixed affinities including translating other IP addresses, 
translating port numbers or dispatching packets to other 
machines. Some of these other actions are described below. 

FIG. 4 is a diagram illustrating a network that includes 
two forwarding agents and two service managers. A first 
client 402 and a second client 404 send packets through a 
network or internetwork 406 that eventually reach a subnet- 
work that includes a first forwarding agent 410, a second 
forwarding agent 412, a first service manager 420, and a 
second service manager 422. In the examples shown, the 
service managers communicate with the forwarding agents 
and with each other over the same physical network that is 
used to send packets. In other embodiments, a separate 
physical connection may be provided between service man- 
agers for the purpose of coordinating service managers and 
providing back up service managers and a separate connec- 
tion may be provided between the service managers and the 
forwarding agents for the purpose of multicasting wildcard 
affinities or, in some embodiments, for sending fixed affini- 
ties and returning packets to forwarding agents. 

In general, the service managers may communicate 
amongst themselves and with the forwarding agents in any 
manner appropriate for a particular system. The forwarding 
agents each are connected to a first server 430, a second 
server 432 and other servers up to an nth server 440. These 
servers may represent one or more virtual machines. Packets 
from the clients may be routed through either forwarding 
agent 410 or forwarding agent 412. In fact, packets corre- 
sponding to the same connection or flow may be routed at 
different times through different forwarding agents. To cope 
with this situation, the service managers multicast wildcard 
affinities to both forwarding agents. When either forwarding 
agent first receives a packet for a flow, that forwarding agent 
forwards the packet to the manager that has requested the 
packet using a wildcard affinity so that the service manager 
can provide the forwarding agent with the fixed affinity that 
defines how to handle the packet. 

FIG. 5 is a diagram illustrating how a service manager 
provides instructions to two separate forwarding agents for 
handling a connection. A client 500 sends a SYN packet to 
a first forwarding agent 502. Forwarding agent 502 has 
previously received a wildcard affinity from a service man- 
ager 504 on a dedicated connection on which service man- 
ager 504 multi casts wildcard affinities to forwarding agents. 
As a result of the wildcard match, forwarding agent 502 
encapsulates the SYN packet and forwards it to service 
manager 504. Service manager 504 receives the SYN packet 
and returns it to forwarding agent 502 along with a fixed 
affinity specifying an action to be performed on the packet. 
The action defined in this example is translating the desti- 
nation IP address of the packet from a virtual IP address to 
the IP address of a host 506. Hosts 506 and 507 together 
implement a virtual machine 510. 

Host 1 receives the SYN packet from forwarding agent 1 
and returns a SYN ACK packet back to client 500. However, 
for some reason, the SYN ACK packet from host 1 is routed 
not through forwarding agent 502, but instead through 
forwarding agent 512. Forwarding agent 512 receives the 
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SYN ACK and notes that it matches a wildcard affinity 
corresponding to the flow of packets from host 506 to client 
500. Forwarding agent 512 encapsulates the SYN ACK 
packet and sends it to service manager 504. Service manager 

5 504 defines an action for the SYN ACK packet and includes 
that action in a second fixed affinity which it sends along 
with the encapsulated SYN ACK packet back to forwarding 
agent 512. Forwarding agent 512 then sends the SYN ACK 
packet on to client 500 where it is processed. 

10 At this point, forwarding agent 502 has a fixed affinity for 
the flow from client 500 to the hosts and forwarding agent 
512 has a fixed affinity for the flow from the hosts back to 
client 500. Each forwarding agent continues to handle flows 
without fixed affinities using the wildcard affinities. The 

15 service manager acts as a point of synchronization between 
the forwarding agents when the forwarding agents handle 
common flows. 

Client 500 then sends a data packet which happens to be 
routed through forwarding agent 512 and not forwarding 

20 agent 502. Forwarding agent 502 has received the fixed 
affinity that provides instructions on how to deal with 
packets in the flow from client 500 to virtual machine 510. 
However, forwarding agent 512 has not yet received that 
fixed affinity. Forwarding agent 512 has received a wildcard 

25 affinity previously multicast by the service manager. 
Therefore, forwarding agent 512 detects a wildcard affinity 
match for the data packet and encapsulates the data packet 
and sends it to service manager 504. 

3Q Service manager 504 receives the data packet and notes 
that the data packet matches the previously defined first 
fixed affinity which was sent to forwarding agent 502. 
Service manager therefore does not run the load balancing 
algorithm again to determine where to route the data packet, 

35 but instead returns the first fixed affinity to forwarding agent 
512 along with the data packet. Forwarding agent 512 
receives the data packet and the fixed affinity and then has 
the same instructions as forwarding agent 502 for handling 
that data packet and other packets in the flow from client 500 

40 to virtual machine 510. Forwarding agent 512 therefore 
translates the destination IP address of the data packet to the 
IP address of host 506 and forwards the packet on to host 
506. 

Thus, as long as wildcard affinities are received by each 

45 forwarding agent, the service manager is able to provide 
fixed affinities to each forward agent whenever a fixed 
affinity is required to provide instructions to handle packets 
for a given flow. Once a fixed affinity is defined for a flow, 
the same fixed affinity is provided to any forwarding agent 

so that returns a packet to the service manager as a result of a 
wildcard match. 

To provide a load balancing service for HTTP, a service 
manager sends a pair of wildcard affinities (one for each 
direction of flow to and from a virtual machine) to a 

55 multicast group that includes each available router in a 
network. The wildcard affinities specify a protocol and also 
indicate an exact match on the IP Address and HTTP port 
number for the virtual machine and an IP address and mask 
combination that identifies the client population that is 

60 serviced by the service manager. The client population 
serviced by the service manager is referred to as the client 
domain of the service manager. If multiple service managers 
are used, then each service manager may be configured to 
service a different client domain. 

65 For example, if the majority of traffic is comiog from a 
small number of firewalls, whereby the same foreign IP 
address is shared by many different clients, all those affini- 
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ties can be assigned by one service manager. Thus, traffic 
from large sites can be isolated from other traffic and 
assigned to a different service manager. 

Thus, the architecture is scalable and service managers 
may be added to handle client domains as needed. The set of 
clients serviced by each service manager can be changed by 
canceling the wildcards that each service manager has 
broadcast to forwarding agents and sending new wildcards 
specifying the new client domain. 

When multiple service managers are included, it is impor- 
tant that the client domains specified by service managers 
performing the same service do not overlap. The task of 
assigning affinities for each client domain is centralized by 
the service manager serving that domain so all packets for a 
given flow are controlled by a single service manager. For 
example, if duplicate SYN packets are sent by a client, both 
should be directed to the same service manager and assigned 
the same fixed affinity. If the packets were directed to 
different service managers, then the service manager load 
balancing algorithms might assign different real machines to 
handle the connections as a result of the network being in a 
different state when the second SYN packet arrived. In 
addition, UDP unicasts from the same client must be 
assigned the same affinity and related connections (e.g., FTP 
control and data connections) must be assigned the same 
affinity. 

Once the forwarding agents have received fixed affinities, 
packets intercepted that match a fixed affinity are processed 
as instructed in the set of actions specified in the fixed 
affinity. If a matching fixed affinity is not found, the packet 
is compared against the wildcard affinities to find manager 
(s) that are interested in this type of packet. If no appropriate 
Wildcard Affinity is found, normal IP routing occurs. 
Generally, a manager uses the wildcard affinity to be 
informed of flows it may be interested in. Once a manager 
has determined bow a flow should be handled, it usually 
sends a fixed affinity so that the processing of subsequent 
packets.for that flow can be offloaded to the forwarding 
agent. In some cases actions for certain flows can be 
predetermined by the service manager without seeing pack- 
ets from the flow. In such cases, the actions may be specified 
in a wildcard affinity and no message need be sent to the 
service manager and no fixed affinity need be generated. The 
service manager may specify that it is still to receive certain 
packet types after a fixed affinity is sent by including an 
optional action interest criteria message segment with the 
fixed affinity. 

In the load-balancing case, a fixed affinity is used to 
identify the server that is to receive this particular flow 
whereas a wildcard affinity is used to define the general class 
of packets for which load balancing is to be performed (all 
those matching the cluster address and port number for the 
clustered service) and to identify the manager that is to make 
the balancing decision for flows that match the wildcard 
affinity. 

Fixed Affinities 

FIG. 6 is a diagram illustrating a fixed affinity 600. Fixed 
affinity 600 matches only one flow through a network. As 
described above, a flow is defined by an affinity key, which 
is a unique 5-tupIe that spans the packet headers: 

IP Header: 

Protocol Type (e.g., UDP or TCP) 
Source IP Address 
Destination IP Address 
TCP or UDP Header: 
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Source Port 
Destination Port 

It should be noted that if the protocol being used is not 
TCP or UDP, then the ports in the affinity key may be set to 
5 0. 

Fixed affinity 600 includes an affinity key 602. In 
addition, fixed affinity 600 contains information that dictates 
how a forwarding agent is to process packets that match the 
affinity key, and how the forwarding agent is to manage the 
10 affinity. 

A dispatch flag 604 indicates whether the packet is to be 
dispatched to the forward IP address included in the fixed 
affinity. Setting the dispatch flag indicates that the packet is 
to be forwarded to a forward IP address 608 that is provided 
in the fixed affinity. The difference between dispatched and 

15 directed traffic is that dispatch traffic is forwarded directly 
from a forwarding agent to a specific server without trans- 
lating the destination IP address of the packet. In other 
words, if a packet is dispatched, then the packet destination 
address is not used to forward the packet. Instead, a for- 

20 warding address contained in an affinity is used to forward 
the packet. If the connection is not dispatched but directed 
by the forwarding agent, then the packet IP destination must 
be translated using NAT if the packet is redirected to a 
specific server. 

2s If forward IP address 608 is zero, then the packet is 
dropped after processing statistics as indicated by an infor- 
mation flag 606. Not setting the dispatch flag indicates that 
the packet is to be forwarded based on the address provided 
in the packet IP header. 

30 Information flag 606 indicates whether or not statistics are 
to be gathered for packets forwarded using the fixed affinity. 
If the Information flag is set, statistics are updated for the 
forward IP address. In one embodiment, the statistics kept 
include: 

35 1. total bytes for all packets matching t he forward P 
address 

2. total packets matching the forward P address 
Statistics for packets and bytes matching the affinity may 
be kept regardless of the setting of the Information flag. 

40 Fixed affinity 600 also includes a time to live 610. Time 
to live 610 specifies the number of seconds before the fixed 
affinity should be timed-out from a fixed affinity cache 
maintained by a forwarding agent. If a time to live of 0 is 
specified, then that means that the fixed affinity is not to be 

45 cached by a forwarding agent and if a copy of the fixed 
affinity is already in the cache, it should be removed. Thus, 
service managers may remove fixed affinities that they have 
sent to forwarding agents by simply sending copies of those 
fixed affinities to the forwarding agents with time to live set 

so to 0. 

Each fixed affinity sent by a service manager is correlated 
to a wildcard affinity previously sent by the service manager. 
If a forwarding agent receives a fixed affinity for which no 
supporting wildcard affinity is found, the forwarding agent 
55 ignores the fixed affinity and discards it. 
Wildcard Affinities 

FIG. 7 is a diagram illustrating a wildcard affinity 700. 
Wildcard affinity 700 is a more general form of Affinity that 
is used by a service manager to register filters with the 

60 forwarding agent(s) that define the range of flows that are of 
interest to the service manager. Like a fixed affinity, wildcard 
affinity 700 also includes a dispatch flag 702 and an infor- 
mation flag 704. Wildcard affinity 700 also includes the 
elements of an affinity key (protocol 706, source IP address 

65 708, destination IP address 712, source port 716, and des- 
tination port 718) plus source netmask 710 and destination 
netmask 714. 
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The netmasks and the source and destination IP addresses 
are used to specify ranges of addresses covered by the 
wildcard affinity. The source netmask is ANDed with the 
source IP address in the wildcard affinity. The source net- 
mask is also ANDed with the source IP address from the 
packet. If the results of (he two operations are equal, then the 
source IP address of the packet is considered to be in range 
of the wildcard affinity. Likewise, the destination netmask is 
ANDed with the destination IP address in the wildcard 
affinity. The destination netmask is also ANDed with the 
destination IP address from the packet. If the results of the 
two operations are equal, then the destination IP address of 
the packet is considered to be in range of the wildcard 
affinity. If both the source and the destination IP addresses 
of the packet are in the range of the wildcard affinity, and the 
ports and protocols also match, then the packet is said to 
match the wildcard affinity. It should also be noted that, in 
one embodiment, a zero specified for a port or a protocol 
matches all ports or protocols. 

It should be noted that in other embodiments, other 
methods of specifying ranges for the wildcard affinity are 
used. For example, in one alternative arrangement, ranges of 
IP addresses are specified by specifying lower bound and 
upper bound IP addressees. All addresses between the two 
bounds fall within the range of the wildcard affinity. In some 
applications, multiple ranges may be specified. The method 
described above is particularly useful for specifying a single 
address, specifying all addresses in a subnet, or specifying 
every even or odd address, every fourth address, every 
eighth address, etc. 

For example, to specify a single host of 1.1.1.1, the 
wildcard affinity include an IP address of 1.1.1.1 with a 
netmask of 255.255.255.255. To specify the range of hosts 
from 1.1.1.0 to 1.1.1.255, the wildcard affinity would 
include an IP address of 1.1.1.0 with a netmask of 
255.255.255.0, indicating that the first three bytes of the IP 
address must match exactly and that the last byte is to be 
ignored. 

Wildcard affinity 700 also includes a time to live 722. 
Time to live 772 is used in the same manner as the time to 
live for the fixed affinity. Wildcard affinities are deleted by 
forwarding agents based on the time to live set for the 
wildcard affinity by the service manager. The timing of such 
a deletion need not be exact. In one embodiment, the timing 
need only be accurate to within two seconds. This same 
tolerance is for fixed affinities as well. Service managers 
must refresh each wildcard affinity before its time to live 
expires in order to continue to receive packets that match the 
wildcard affinity from forwarding agents. As with the fixed 
affinity, a wildcard affinity may be deleted by sending a 
duplicate wildcard affinity with a time to live of 0. 

Actions 

Thus, fixed affinities specify individual flows and packets 
and wildcard affinities specify sets of flows to be processed 
in a special way. Such processing is defined by associating 
actions with the affinities. Actions defined for the affinities 
specify the service to be performed by the forwarding agent 
on behalf of the Manager. For fixed affinities, services 
specified may include: 

Interest Criteria — a list of packet types that cause a 

notification to be sent to the service manager. 
Sequence Number Adjustment — a set of deltas and initial 
sequence numbers by which the TCP sequence num- 
bers and ACK numbers are to be adjusted. 
NAT — provides details for how Network Address Trans- 
lation is to be performed. 
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For Wildcard Affinities, applicable actions are: 

Interest Criteria — a list of packet types that cause a 

notification to be sent to the service manager. 
Advertise — indicates that the destination IP Address in 
5 the Wildcard Affinity is to be advertised by the for- 
warding agent. This may. be done by including the 
destination IP address in routing protocol updates. 
Sequence Number Adjustment — a set of deltas and initial 
sequence numbers by which the TCP sequence num- 
10 bers and ACK numbers are to be adjusted. 

NAT — provides details for how Network Address Trans- 
lation is to be performed. 
Forwarding agents may not support all possible actions. 
For example, some forwarding agents may not support NAT. 
15 The set of actions that the service manager expects a 
forwarding agent to support are identified in an action list 
which may be included with the wildcard affinity. If the 
forwarding agent does not support one or more of the actions 
identified in the list, it discards the wildcard affinity and send 
20 a message to the service manager indicating that it does not 
support all of the actions in the list. This message is referred 
to as an affinity update deny message. The service manager 
then may attempt to send a new wildcard affinity that 
excludes any unsupported actions identified in the affinity 
25 update deny message. 
Service Messages 

Wildcard affinities, fixed affinities, actions, packets, and 
other messages are sent between service managers and 
forwarding agents encapsulated in service messages. In one 

30 embodiment, messages sent between service managers and 
forwarding agents are sent using the specific service mes- 
sage format described below. Service messages are sent 
between service managers and forwarding agents using 
UDP. Wildcard affinities, which are sent by service 

35 managers, can be multicast to a multicast IP Address and 
UDP Port known to the service manager(s) and forwarding 
agent(s), or can be unicast to a particular forwarding agent 
or service manager. FIG. 8A is a diagram illustrating a 
service message header used in one embodiment. Service 

40 message header 800 includes a protocol version 802 and a 
message type 804. The protocol version identifies the ver- 
sion of the service protocol supported by the sender. The 
message type identifies the overall purpose of this message, 
the base format for the message, and implies the set of 

45 optional message segments that may be included in the 
message. 

The following service message types are used: 

50 

Message Type 

affinity update- wildcard affinity 
affinity update- fbicd affinity 
affinity update -deny 
55 interest match-wildcard affinity 

interest match-fixed affinity 
IP packet only 



The affinity update-wildcard affinity message is used to 
60 send wildcard affinities from a service manager to forward- 
ing agents. The affinity update-fixed affinity message is used 
to send fixed affinities. The affinity update-deny message is 
used to report that an affinity update message has been 
rejected because required actions included in the affinity 
65 update are not supported by the receiver. The interest 
match-wildcard affinity message is used to report a wildcard 
affinity match to a service manager and the interest match- 
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fixed affinity message is used to report a fixed affinity match 
to a service manager. The IP packet only message is used to 
forward an IP packet. 

After the service message header, a service message 
includes one or more message segments. Each message 5 
segment begins with its own segment header. FIG. 8B is a 
diagram illustrating a segment header. Segment header 810 
includes a Required Sag 812. Required flag 812 defines 
whether the sender will allow the rest of the message to be 
processed even if the segment cannot be processed (either 10 
because the receiver does not support the function described 
by the segment or because the receiver does not understand 
the segment). The required flag either indicates that the 
segment may be ignored or that the segment is required. If 
a required segment cannot be processed, then the entire 15 
message that includes the segment is dropped and an error 
message is returned to the sender. Each segment header is 
followed by data that is specific to the message segment. 

The following message segments are used: 

20 



Segment Name 



Wildcard Affinity 
Fixed affinity 
Affinity Interest 
Service Precedence 
Security 

Service Manager Interest Data 
forwarding agent Interest Data 
Identity Info 
Action-tVAT 
Actio n- Advertise 
Action-Sequence Number Adjust 
Action- Interest Criteria 
Action List 
IP Packet 



25 



30 



35 



The fixed affinity, wildcard affinity and security segments 
are described immediately below. The remaining segments 
are described in detail following a description of the mes- 
sage types that include the segments. 40 

Security 

If security is expected by the receiver, a security message 
segment immediately follows the service message header. 
The security message segment contains the expected secu- 
rity sequence. If the receiver does not expect security, the 45 
security message segment is ignored (if present) and the 
message is accepted. Security is generally not required for IP 
packet only messages. If authentication is successful, the 
signals are accepted. If the authentication fails, the signal is 
ignored. Various authentication schemes such as MD5 may 50 
be supported. The type of authentication to be used is 
configured at the senders and receivers, along with a pass- 
word. If the receiver does not expect authenticated 
messages, then the security segment may be ignored if it is 
present and the signal may be accepted whether or not it 55 
contains a security segment. 

FIG. 8C is a diagram illustrating a security message 
segment. Security message segment 820 includes a security 
type field and a security data field 824. Security type field 
822 describes the type of encoding used for security (i.e., 60 
MD5, etc.). Security data field 824 contains the data needed 
to implement the algorithm identified by the security type 
field 822. 

Detailed Message Descriptions 
Wildcard Affinity Update 65 
FIG. 9A is a diagram illustrating an affinity update wild- 
card message. Affinity update wildcard message 900 is sent 



by a service manager to a forwarding agent to register or 
unregister for classes of flows that match the specified sets 
of flows. It includes a service message header 902 followed 
by a sequence of message segments. A security segment 903 
is optional, as dictated by the needs of the receiver. A 
wildcard affinity segment 904 is required, since the purpose 
of the affinity update wildcard message is to send a wildcard. 
An action list segment 906 is optional. Its purpose is list the 
actions that a forwarding agent must support in order to 
receive the affinity. If the forwarding agent determines that 
any of the actions are not supported, then it may send an 
affinity update deny message to the service manager. 

An affinity service precedence field 908 is optionally used 
to specify the precedence of the service being provided. This 
allows multiple service managers or a single service man- 
ager to send wildcard affinities for different services. An 
affinity backup precedence field 909 is also optionally used 
to specify the backup precedence of the service manager that 
sent the affinity. This allows a backup service manager to 
send wildcard affinities that are ignored until a higher 
backup service precedence wildcard affinity that corre- 
sponds to a primary service manager is deleted. An identity 
information segment 910 is optionally used to identify the 
manager. This information may be used, for example, in an 
error message on the console of the forwarding agent to 
indicate which service manager had a problem. A service 
manager interest data segment is optionally used to include 
data that should be returned to the service manager when an 
interest match-wildcard affinity message is sent to the ser- 
vice manager as a result of a forwarding agent determining 
a wildcard affinity match. Finally, one or more action 
segments are optionally included. The action segments 
specify actions that are performed on the packets for the 
purpose of providing a network service. It should be noted 
that in some embodiments, fields which are described above 
as optional may become required and required fields may be 
optional. This is also generally true of the other message 
descriptions contained herein. 
Fixed Affinity Update 

FIG. 9B illustrates a fixed affinity update message that is 
sent by a service manager to a forwarding agent to add a 
fixed affinity to the receiver's affinity cache or delete a fixed 
affinity that is stored in the receiver's affinity cache. If the 
time to live in the fixed affinity segment is non-zero, the 
affinity is added to the cache (or refreshed, if it already 
resides there) for the number of seconds specified in the time 
to live. If time to live is zero, the fixed affinity is removed 
from the cache if it is found there. 

Fixed affinity update message 920 includes a service 
message header 922. An optional security segment 924 is 
included as dictated by the needs of the receiver. A fixed 
affinity segment 926 includes the fixed affinity being sent. 
An affinity service precedence 928 optionally specifies a 
service precedence. An affinity backup precedence field 929 
is also optionally used to specify the backup precedence of 
the service manager that sent the affinity. This allows a 
backup service manager to send affinities that arc ignored 
until a higher backup service precedence affinity that cor- 
responds to a primary service manager is deleted. One or 
more action segments 930 are optionally included to specify 
actions to be performed by the receiver for matching pack- 
ets. An identity information segment 932 is optionally used 
to identify the service manager that sent the fixed affinity. A 
service manager interest data segment 934 is optionally used 
to include data that should be returned to the service 
manager when an interest match-wildcard affinity message 
is sent to the service manager as a result of a forwarding 
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agent determining a wildcard affinity match. A forwarding segment 964 is included as dictated by the needs of the 

agent interest data segment 936 is optionally used to include receiver. An affinity identifier segment 966 includes the 

data that a forwarding agent requested to be returned to it affinity key of the affinity that caused the match, the dispatch 

along with a fixed affinity. Finally, an IP packet segment 938 and information flags of that affinity, and an interest match 

includes an IP packet. 5 ne ^ that provides reasons from the interest criteria that 

Usually, the IP packet segment is an IP packet that was caus ^ d lhe match - In one embodiment, a bit vector is used to 

sent to a service manager as a result of a wildcard affinity provide the reasons. 

match and that is being sent back to a forwarding agent ^ ldcntltv information segment 968 is optionally 

along with actions to be performed for the packet. In many included from the original affinity update that prompted the 

implementations, the forwarding agent does not devote 10 interest match message to be sent. Aservice manager interest 

resources to storing packets that have matched a wildcard data 970 * optionally used to include data that the 

affinity and have been forwarded to a service manager. t*™** manager requested when an interest match message 

Therefore, the forwarding agent sends the packet to the fe ^ 10 the Mivice manager. A forwarding agent interest 

service manager along with an interest match message and data segment 972 is optionally used to include data that a 

the service manager sends the packet back to the forwarding is forwarding agent requested to be returned to it along with a 

agent with a fixed affinity update. Thus, the service manager fixed aEnitv - Finally, an IP packet segment is optionally 

stores the packet for the forwarding agent and returns it to included so that the forwarding agent can send the IP packet 

the forwarding agent when the forwarding agent needs to mat caused thc affimtv match t0 thc x ™ cc manager. The IP 

execute an action on the packet. This eliminates the need for P ackct 15 ^ tf thc corresponding data flag in the mterest 

storage and garbage collection at the forwarding agent for 20 catena indicated that the IP Packet should be sent. The IP 

packets that matched a wildcard affinity and are awaiting P acket mav be ^ as a segment of the interest match 

instructions from a service manager for handling. In some message or may be forwarded independently in a subsequent 

implementations, the forwarding agents may temporarily IP Packet message, depending on the capabilities of the 

store packets that have matched a wildcard affinity. forwarding agent. 

However, it has been found that sending packets to the 25 IP Packet 0nlv 

service manager and having the service manager return F1G 9E is a illustrating an IP packet only 

packets with fixed affinities simplifies and improves the message. IP packet only message 980 is sent by a forwarding 

performance of the forwarding agent. a 6 ent to a t*™™ manager or vice versa whenever an IP 

Affin ity Update-deny network packet is sent from one to the other. This can occur 

FIG. 9C is a diagram illustrating an affinity update-deny 30 m a number of situations, e.g.: 

message. An affinity update-deny message is sent by the (1) When a forwarding agent needs to send a service 

forwarding agent to a service manager when the forwarding manager a packet that could not be included with an 

agent receives an affinity update with a required segment interest match message. 

that it cannot process (one where the 'Required* flag is set (2) When a forwarding agent needs to send a service 

either within the segment header or within the list of 35 manager a packet that matched a service manager 

segment types from the action list, if one was included). The wildcard affinity. 

segments that cannot be processed properly are identified in (3) When a service manager needs to send a forwarding 

the action list that is returned with the affinity update-deny agent a packet that it has processed and that needs to be 

message. forwarded to the next appliance (or, if there are no other 

Affinity update-deny message 940 includes a service 40 appliances, to its correct destination). Encapsulating IP 

message header 941. An optional security segment 942 is packets in the IP packet only message avoids loops in 

included as dictated by the needs of the receiver. An action the system by signaling the forwarding agent that the 

list segment 944 includes actions that are not supported by packet has already been to the manager and need not be 

the forwarding agent and that caused the forwarding agent to sent there again. 

sent the affinity update-deny message. An affinity segment 45 IP packet only message 980 includes a service message 

946 from the original affinity update that prompted the header 982. An IP Packet segment 984 includes the IP 

affinity update-deny message is optionally included. An packct. Preferably IP packet only message 980 does not 

identity information segment 948 is from the original affinity include a security segment, since the flow is essentially just 

update that prompted the affinity update-deny message is another IP hop and faster forwarding can be achieved 

also optionally included. A service manager interest data 50 without a security segment 

segment 950 is optionally used to include data that the The messages sent between forwarding agents and service 

service manager sent to the forwarding agent for the for- managers have now been described in some detail. The 

warding agent to send back to the service manager when an wildcard affinity segment, the fixed affinity segment, and the 

interest match-wildcard affinity message is sent to the ser- security segment have also been described. The remaining 

vice manager. The service manager interest data is used by 55 message segments are described in greater detail below in 

the service manager to help process the message. A forward- connection with FIGS. 10A through 101. It should be noted 

ing agent interest data segment 952 is optionally used to that each segment includes, in addition to the fields that are 

include data that thc forwarding agent requests to be shown, a segment header. 

returned to it along with a fixed affinity. FIG. 10A is a diagram illustrating an affinity identifier 

Interest Match (Wildcard affinity or Fixed affinity) 60 segment. Affinity identifier segment 1000 includes a dis- 

FIG. 9D is a diagram illustrating an interest match mes- patch flag 1002, an information flag 1004, and an affinity key 

sage for either a wildcard affinity or a fixed affinity. Interest 1006. These fields are defined the same as they are defined 

match message 960 is sent by the forwarding agent to a for fixed affinities and wildcard affinities. Affinity identifier 

service manager when an IP packet matches the interest segment 1000 also includes an interest mask 1008 that 

criteria that was sent the last time the matching affinity was 65 provides reasons from the interest criteria sent by the service 

refreshed or added in the cache. Interest match message 960 manager that caused the match. This gives the service 

includes a service message header 962. An optional security manager notice of what affinity caused the match and also 
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what interest criteria in that affinity caused the match. The 
interest criteria action specified in an affinity sent by a 
service manager is described further below. 

FIG. 10B is a diagram illustrating an affinity service 
precedence segment. Affinity service precedence segment 
1010 includes a search order flag 1012 that specifies the 
search order for the precedence, i.e., whether a higher 
priority precedence is represented by a higher or a lower 
priority number. A precedence value field 1014 actually 
provides the precedence value. The service precedence 
enables one or more service managers to provide different 
services that are executed in sequential order based on the 
precedence values provided. In this manner, multiple affini- 
ties may be specified that match a flow, with each affinity 
corresponding to a different service that specifies different 
actions to be performed for packets in the flow. A packet for 
such a flow may be forwarded to several service managers 
before it is eventually sent to the client or the specific server. 
It should be noted that only the last service manager can 
dispatch the packet since the packet must be returned by 
higher priority service managers to the forwarding agent for 
further processing by lower priority service managers. 

Thus, the affinity service precedence allows multiple 
service managers of different types to control the same flow. 
The value of the precedence dictates the order in which the 
forwarding agent should process affinities if multiple 
matches occur. When a matching affinity contains an action 
that requires the packet to be sent to a service manager, the 
action is honored. When the packet is returned, the forward- 
ing agent processes the affinity contained in the response and 
continues with the matching affinity of the next highest 
precedence. 

FIG. 10C is a diagram illustrating a service manager 
interest data segment. Service manager interest data segment 
1020 includes an interest data field 1021 that can contain 
anything that the service manager arbitrarily determines. 
This is simply data that can be sent by the service manager 
to the forwarding agent The forwarding agent returns the 
data to the manager with an interest match message when an 
interest match is determined. Typically, this data is used to 
index the affinity. 

FIG. 10D is a diagram illustrating a forwarding agent 
interest data segment. Forwarding agent interest data seg- 
ment 1022 includes an interest data field 1023 that can 
contain anything that the forwarding agent arbitrarily deter- 
mines. This is simply data that can be sent by the forwarding 
agent to the service manager when an interest match is sent 
to the service manager. The service manager returns the data 
to the forwarding agent with any fixed affinity update 
message that is sent as a result of the interest match. 
Typically, this data is used to index the affinity. 

FIG. 10E is a diagram illustrating an identity information 
segment that is used to identify the sender of a service 
message. The identity information may be used for logging 
and debugging. Identity information segment 1024 includes 
an IP address field 1025 that contains the IP address of the 
message sender. A character field 1026 contains the name of 
the host. 

FIG. 10F is a diagram illustrating a NAT (Network 
Address Translation) action segment. NAT action segment 
1030 includes fields that specify a source IP address 1032, 
a source port 1034, a destination IP address 1036, and a 
destination port 1038 that are to replace the corresponding 
fields in the packet. The NAT action segment thus specifies 
that NAT is to be performed on any packet that matches the 
associated affinity. A NAT action segment can be included 
with any Wildcard or Fixed affinity sent by a service 
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manager to a forwarding agent. The action is not performed 
on packets that are forwarded to the service manager. If the 
packet is forwarded to the service manager, then the packet 
is not immediately altered. If the service manager sends the 

5 packet back to the forwarding agent for- forwarding, the 
action is performed by the forwarding agent at that time, 
therefore removing the need for the manager to implement 
that function directly. 

FIG. 10G is a diagram illustrating a sequence number 

10 adjust action segment. Sequence number adjust action seg- 
ment 1040 specifies that a forwarding agent should adjust 
sequence numbers and ACK numbers in. the TCP packets 
that match the associated affinity. A sequence number adjust 
action segment can be included with any wildcard affinity or 

15 fixed affinity sent by a service manager. The sequence 
number adjust is not performed on packets that are for- 
warded to the service manager. The action may be per- 
formed when the service manager returns the packet back to 
the forwarding agent for forwarding. 

20 A sequence delta field 1042 specifies the amount by which 
the sequence number in packets is to be adjusted. An initial 
sequence number 1044 specifies the lowest sequence num- 
ber to which the delta is to be applied. An ACK delta field 
1046 specifies the amount by which to adjust the ACK 

25 number. An Initial ACK number field 1048 specifies the 
lowest ACK number to which ACK Delta is to be applied. 
Thus, sequence numbers and ACK numbers in packets can 
be modified by forwarding agents according to a scheme 
determined by a service manager. The scheme is sent to the 

30 forwarding agents using the sequence number adjust action 
segment. 

FIG. 10H is a diagram illustrating an advertise action 
segment. An advertise action segment is sent by a service 
manager to a forwarding agent to specify that the destination 

35 IP address in an enclosed wildcard affinity is to be advertised 
by the forwarding agent. That means that the address is 
included in routing protocol updates, just as if the destina- 
tion IP address belonged to a device connected to the router. 
The address advertisement is deleted when the associated 

40 wildcard affinity is deleted. By directing a forwarding agent 
to advertise an address, the service manager can simulate the 
presence of an network service appliance at the location of 
the forwarding agent. For example, if the service manager is 
providing load balancing among a group of hosts, the service 

45 manager would direct a forwarding agent to advertise the 
virtual IP address of the cluster of hosts. Thus, the virtual IP 
address can be advertised as if a load balancer at the location 
of the forwarding agent were advertising the virtual IP 
address. If a forwarding agent receives a packet destined for 

50 the advertised address, but that packet does not match an 
affinity (either Full or Wildcard), the packet is dropped. This 
avoids establishing connections to the forwarding agent for 
ports that no service manager is supporting. 

Advertise action segment 1050 includes an advertise 

55 address 1052, which is the address to be advertised by the 
forwarding agent. A subnet mask 1054 may also be used for 
such advertising. If a subnet mask is used, then the IP 
address and mask combination indicates a subnet to be 
advertised. The advertise segment can also be used without 

60 specifying a subnet mask. 

FIG. 101 is a diagram illustrating an interest criteria 
action. Interest criteria action 1060 is sent by a service 
manager to a forwarding agent to specify that the service 
manager is to be informed when certain types of special 

65 packets are detected by the forwarding agent. Interest cri- 
teria action 1060 includes an interest IP address 1062 and an 
interest port 1064. The interest IP address and port specify 
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an IP address and port to which the interest match message does not specify that the actions arc to be performed. Its 

is to be sent. An interest mask 1066 is bit vector that purpose is to warn the forwarding agent of the service 

specifies the types of packets for which the service manager requirements. The forwarding agent responds with an affin- 

is requesting notification. The type of packet specified by the ity update-deny and discards a wildcard affinity if the 

bits may be a function of the protocol type specified in the 5 forwarding agent cannot support all the actions in an action 

affinity encapsulated with the interest criteria action. For list that is provided with the wildcard affinity. Action list 

example if the protocol is TCP, then in one embodiment, the segment 1070 includes a first action type 1072. Action list 

bits are interpreted as follows: segment 1070 may also include a second action type 1074 

Bit 0=1:: FIN afl d omer action types up to an nth action type 1080. 

Bit 1=1:: SYN 10 A service message protocol for sending messages and 

Bit 2-1 ■• RST packets between service managers and forwarding agents 

ii ncu Das * jeeD defined to FIGS. 6-10J. Each service message 

Bit 3=1.: PiH includes a service message header that identifies the mes- 

Bit 4=1:: ACK sa g e type. After the service message header, each service 

Bit 5-1:: URG 15 message .includes one or more segments, depending on the 

Bit 6=1:: Data Present message type. Each segment begins with a segment header. 

Bit 7=1:: First Data present Using the message types described, service managers can 

Bit 8=1:: Fragmented packet, and the source/destination ^ forwarding agents instructions detailing certain sets of 

IP addresses match packets that the service manager wants to either to be 

R't IS 1- All P t t 20 f orwar ded to the service manager or to cause an interest 

iAl '\ 1 -^ffrfo .u .u u-« - . j match message to be sent to the service manager. Messages 

If the Drotocol is UDr, then the bits are interpreted as , . ., . , f . . 

follows F are also used to specify actions for certain packets in certain 

flows. 

Bit 6=1:: Data Present For exam pi ej jf a service manager is providing load 

Bit 7=1:: First Data present 2 5 balancing, the service manager first sends a wildcard affinity 

Bit 8=1:: Fragmented packet, and the source/destination update message to a forwarding agent specifying a set of 

IP addresses match clients that the service manager will load balance. The 

Bit 15=1:: All Packets wildcard affinity may also include an action that directs the 

For other protocols, Bit 15 may be set to indicate all forwarding agent to advertise a virtual IP address for a 

packets. 30 virtual machine that includes all of the load balanced serv- 

A data flag 1067 uses the same bit code as the interest ers. When the forwarding agent intercepts a packet that 

mask. Whereas the interest mask determines whether the matches the wildcard affinity, then the forwarding agent 

service manager should be forwarded an interest match sends an interest match message to the service manager. The 

message, data flag 1067 specifies whether the service man- service manager then determines a server to assign the 

ager is to receive a copy of the packet that caused the interest 35 connection (or the server that has already been assigned the 

match with the interest match message. If a bit is set, then connection) and sends a fixed affinity to the forwarding 

the forwarding agent is to send the packet as well as the agent that directs the forwarding agent to dispatch the packet 

interest match to interest IP address 1062 and interest port to that server or to use NAT to substitute the server's address 

1064. It should be noted that in some embodiments, the in the packet. The service manager also may include an 

forwarding agents may send messages and forward packets 40 interest criteria in a fixed affinity that specifies that future 

to service managers over a different network so that the packets for the flow should not be sent to the service 

interest IP address and interest port may not be used or some manager, but that the service manager should be notified if 

other method may be used for specifying where interest certain types of packets such as a FIN or a FIN ACK are 

match messages and packets should be sent to the service received. At any point, the service manager may cancel a 

manager. 45 fixed affinity or a wildcard affinity sent to a forwarding agent 

A copy flag 1068 also uses the same bit code as the by sending a fixed afEnity or a wildcard affinity with a time 

interest mask. Each bit specifies whether a copy of the to live of 0. 

matching packet is to be forwarded to the server. If the bit Thus service managers are able to control affinities and 

is set for the packet type, the forwarding agent sends a copy monitor flows using the above defined messages. When a 

of the matching packet and refers to a hold Gag 1069 to 50 forwarding agent receives a packet, affinities received from 

determine what to do with the original packet. Hold flag service managers are searched first for the one with the 

1069 also uses the same bit code as the interest mask. Hold highest service precedence. Once a match is determined, the 

flag 1069 determines whether the forwarding agent forwards search order defined for that precedence is used to find 

the packet to the service manager or, if possible, holds the another identical Affinity with a better service precedence. If 

packet and waits for the service manager to send a fixed 55 multiple affinities exist with the same best service 

affinity that specifies how the packet should be forwarded by precedence, they are searched for the one with the lowest 

the forwarding agent. If the bit is not set for the packet type, backup precedence value. 

then the forwarding agent forwards the packet. If the bit is Service managers manage the storage of affinities on 

set, then the forwarding agent holds the packet, if possible. forwarding agents using the time to live portion of the 

If the packet cannot be held by the forwarding agent for 60 affinity segments. The forwarding agents remove affinities at 

some reason (e.g., lack of storage) then the forwarding agenl intervals specified by the service manager if they have not 

forwards the packet to the Manager. already been removed at the request of a manager (via an 

FIG. 10J is a diagram illustrating an action list segment. affinity update message with a time-to-live of zero). No 

Action list segment 1070 is sent by a service manager to a affinity is kept for an interval longer than the interval 

forwarding agent with wildcard affinities to specify all the 65 specified by the time-to-live set by the manager (within a 

actions that must be supported in order for the forwarding tolerance of +/-2 seconds in one embodiment) so that the 

agent accept the wildcard affinity. Action list segment 1070 manager can reliably assume that the affinities have been 
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cleared at some small time beyond that interval that accounts 
for any propagation or processing delays. This simplifies the 
managing of affinities by the service manager across mul- 
tiple routers. In some cases, a forwarding agent may need to 
ask for an affinity again if more traffic arrives for that affinity 5 
after it has been deleted. 

The service manager itself stores affinities long enough to 
allow forwarding agents sufficient time to delete their own 
copies. If an affinity is allowed to expire at a service 
manager, it must be kept by the service manager long 10 
enough so that the forwarding agents have deleted their 
copies first. This avoids mismatches of affinities across 
routers should a new affinity assignment request be received 
while a router still has the old affinity. 

Service managers also keep affinities long enough after an is 
outbound FIN is detected for a connection so that the final 
inbound ACK (or in the case of many Windows web 
browsers, the inbound RST) can be forwarded to the appro- 
priate host. The use of a ' sticky* timer at the service manager 
satisfies this requirement. If a service manager changes an 20 
affinity at a time when it is possible that the affinity is still 
cached by a forwarding agent, the service manager asks the 
forwarding agents to delete the affinity before sending the 
updated affinity. 

It should be noted that fixed affinities and wildcard 25 
affinities do not themselves include actions in the data 
structures described above. For flexibility, actions are 
defined separately but are included with fixed affinities or 
wildcard affinities in an affinity update message. The asso- 
ciated actions are stored along with the fixed affinity or 30 
wildcard affinity on service managers and forwarding 
agents. Whenever a fixed affinity or a wildcard affinity is 
referred to as being stored on a forwarding agent or a service 
manager, it should be understood that associated actions 
may be stored with the affinity, whether or not such actions 35 
are explicitly mentioned. 

Likewise, other items may be included in a stored affinity 
data structure. For example, the affinity may include a time 
to live when it is sent by a service manager. When the affinity 
is received by a forwarding agent, the forwarding agent may 40 
compute an expiration time from the time to live and store 
the expiration time along with the fixed affinity. 

An architecture that includes service managers and for- 
warding agems for providing network services has been 
disclosed. A message protocol for sending messages from 45 
service managers to forwarding agents and for reporting 
activity and forwarding packets from forwarding agents to 
service managers has been disclosed as well. 

Since the service manager does not need to be located at 
a strategic point in a network, it is possible to include one or so 
more backup service managers at different locations in the 
network. When a primary service manager fails, a backup 
service manager can assume the role of the failed primary 
service manager and provide instructions to forwarding 
agents so that network services can still be provided. 55 

In order for network services to be provided by a backup 
service manager without interruption, the backup service 
manager must receive information about the flows being 
bandied by the primary service manager and store such 
information so that it is ready to provide instructions to the 60 
forwarding agents as soon as the primary service manager 
fails. Without such state information, the backup service 
manager would not be able to immediately begin renewing 
fixed affinities from forwarding agents when those fixed 
affinities expire. Instead, the newly active backup service 65 
manager would need to use its own state machine to deter- 
mine how forwarded packets from the forwarding agents 



should be handled. Each connection formerly being handled 
by the failed service manager would need to be reset and 
reestablished by the newly active service manager. 

In one embodiment, the primary service manager sends a 
replication packet to the backup service manager whenever 
a fixed affinity is sent to a forwarding agent. Traffic between 
the primary service manager and the backup service man- 
ager is minimized by having the backup service manager 
keep track of all such fixed affinities. The fixed affinities 
expire on the backup service manager in the same manner 
that the fixed affinities expire on the primary service man- 
ager. Thus, messages need not be sent between the primary 
service manager and the backup service manager for the 
purpose of timing out fixed affinities. Fixed affinities are 
timed out by the backup service manager using the same 
criteria applied by the primary service manager. 

Likewise, fixed affinity expiration time intervals are 
renewed on the backup service manager upon the receipt of 
a replication packet set by the primary service manager at 
the same time as a fixed affinity update is sent to a forward- 
ing agent. Thus, the maintenance of fixed affinity on the 
backup service manager minors the maintenance of such 
fixed affinities on the primary service manager. The backup 
service manager is thus ready at any point in time to assume 
the duties of the primary service manager. 

FIG. 11 is a block diagram illustrating a distributed 
network service architecture including service managers and 
forwarding agents. Forwarding agents 1102a and 11026 
receive packets from networks 1103d and 1103f> and route 
packets between those networks. A primary service manager 
1104 is connected to the forwarding agents for the purpose 
of providing instructions to the forwarding agents for han- 
dling packets and providing network services. Primary ser- 
vice manager, 1104 is in communication with a backup 
service manager, 1106 via a service manager interface. It 
should be noted that service manager 1106 may be config- 
ured as a backup service manager only or backup service 
manager 1106 may additionally be configured as a primary 
service manager providing services for its own set of flows. 

Service managers are configured by the system adminis- 
trator to request certain packets corresponding to certain 
flows and to specify a backup service priority for the 
requested flows. Significantly, forwarding agents need not 
be configured to receive instructions from particular service 
managers as either primary service managers or backup 
service managers. The forwarding agent simply follows 
instructions contained in fixed affinities and wild card affini- 
ties received from service managers and prioritizes the 
instructions according to the backup precedence specified in 
the affinity update messages received from the service 
managers. 

Both the primary service manager and the backup service 
manager send wild card affinities to forwarding agents that 
specify sets of packets to be sent to the service managers. 
Wild card affinities sent by the backup service manager have 
a lower backup service priority than the wild card affinities 
sent by the primary service manager. As a result, forwarding 
agents forward packets for new flows to the primary service 
manager. The primary service manager generates fixed 
affinities for the new flows. The fixed affinities are given a 
time to live so that they expire on the forwarding agents and 
forwarding agents must periodically forward packets to the 
primary service manager so that the fixed affinities may be 
renewed. 

When a packet for a flow is forwarded to the service 
manager, the service manager resets an expiration time 
interval for the stored fixed affinity on the primary service 
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manager and continues to service that flow. The backup handle existing connections until the connections are fin- 
service manager likewise stores fixed affinities and allows ished and to cease handling new connections when a wild- 
those fixed affinities to expire after an expiration interval card affinity is deleted. 

unless a fixed affinity is received from the primary service In addition to resetting its own fixed affinity expiration 

manager. Thus, the backup service manager maintains a list 5 time in step 1210, the primary service manager also creates 

of fixed affinities that is substantially the same as the list of a replication packet and sends it to the backup service 

fixed affinities maintained by the primary service manager. manager. FIG. 13 is a flowchart illustrating the process for 

The list is maintained by a combination of receiving repli- sending a replication packet to the backup service manager, 

cation packets containing the fixed affinities from the pri- The process starts at 1302 when the fixed affinity expiration 

mary service manager and allowing the fixed affinities 10 time is reset by the primary service manager. In a step 1304, 

received in the replication packets to expire in a similar the primary service manager adds the fixed affinity to a 

manner to the way that the primary service manager allows replication packet. In a step 1306, the replication packet is 

such fixed affinities to expire. sent to the backup service manager. The process ends at 

F!G. 12 is a flow chart illustrating a process executed by 1308. 
a service manager for managing fixed affinities. The process 15 FIG. 14 is a flowchart illustrating a process implemented 
starts at 1202. In a step 1204, the service manager receives on the backup service manager upon the receipt of a repli- 
a message from a forwarding agent. In a step 1206, the cation packet. The process starts at 1402 when the replica- 
service manager determines if a fixed affinity exists. If the tion packet is received. In a step 1404, the backup service 
fixed affinity exists, and assuming for the purpose of this manager extracts the fixed affinity from the replication 
example that the service manager does not determine for 20 packet. Next, in a step 1406, the backup service manager 
other reasons that the fixed affinity needs to be changed, the adds an expiration interval to the current time to derive an 
fixed affinity is sent to a forwarding agent in step 1208. Next, expiration time when the fixed affinity stored on the backup 
the service manager resets the expiration time of the fixed service manager will expire. In one embodiment, the expi- 
affinity stored on the service manager. The process ends at ration interval added on the backup service manager is the 
1212. 25 same expiration interval used by the primary service man- 

If the fixed affinity does not exist, then control is trans- agcr so that fixed affinities on the backup service manger 

ferred to step 1214 and the service managers state machine expire at about the time the corresponding affinities expire 

generates a fixed affinity. It should also be noted that in on the primary service managers. In some embodiments, the 

certain cases the state machine may determine that an expiration interval on the backup service manager is slightly 

existing fixed affinity may need to be changed in which case 30 longer than the expiration interval on the primary service 

the state machine would generate a changed fixed affinity in manager to allow extra time for the replication packet to be 

step 1214. Control is then transferred to a step 1216 and the sent to the backup service manager, 

service manager forwards the fixed affinity to the forwarding Next in a step 1408, the backup service manager deter- 

agent. The process then ends at 1218. mines whether a fixed affinity exists. If a fixed affinity does 

In step 1210, the service manager resets the expiration 35 exist, then control is transferred to a step 1410 and the fixed 

time of a stored fixed affinity as a result of a fixed affinity affinity is replaced and the new expiration time is stored. If 

interest match message being received from a forwarding a fixed affinity does not exist, then control is transferred to 

agent. Thus, fixed affinities automatically expire on both a step 1412 and the backup service manager allocates 

forwarding agents and service managers. Forwarding agents memory and stores the new fixed affinity. Once a fixed 

have fixed affinities renewed when a service manager 40 affinity has been replaced or a new fixed affinity has been 

resends a new fixed affinity with a time to live specified to stored, the process ends at 1416. 

the forwarding agent. Service managers renew their fixed It should be noted that the backup service manager may 

affinities when they receive a fixed affinity interest match also check to be certain that a wild card affinity exists before 

from a forwarding agent. When a fixed affinity interest storing the fixed affinity received in a replication packet. All 

match message is received, a service manager resets the 45 fixed affinities should correspond to wild card affinities. The 

expiration time stored along with the fixed affinity. check may be used to detect errors in replication packets 

Thus, fixed affinities arc maintained on both the service received and may also be used in some embodiments to 

managers and the forwarding agents on a need-to-know- allow different backup service managers to be partitioned to 

basis. If a connection terminates in a nonstandard fashion, backup different sets of flows for a single primary service 

then the service manager will eventually delete its fixed so manager. The wildcard affinities stored on each of the 

affinity because no fixed affinity interest match messages separate backup service managers partitions the backup 

will be received from a forwarding agent indicating to the service managers. The primary service manager can then 

service manager that the connection is active and causing the broadcast replication packets to all potential backup service 

service manager to reset the expiration time of the corre- mangers since unneeded fixed affinities received in replica- 

spondiog fixed affinity. The forwarding agents will delete 55 tion packets are not stored by backup service managers that 

their copies of the fixed affinity because they will not receive do not also have a corresponding wildcard affinity, 

fixed affinity update messages from the service manager. A system for maintaining state information on a backup 

Affinities may also be deleted explicitly by a service service manager about flows controlled by a primary service 
manager by sending a copy of the fixed affinity to a manager has been disclosed. For new flows, the primary 
forwarding agent with a time to live of zero. Fixed affinities 60 service manager sends a replication packet that includes a 
are also deleted on a forwarding agent when a wildcard new fixed affinity to be stored on the backup service man- 
affinity is sent to the forwarding agent that has a time to live ager. When a forwarding agent sends a packet to the primary 
of zero and that corresponds to the fixed affinity. In addition, service manager corresponding to an existing flow, the 
in some embodiments, fixed affinities may automatically be expiration interval for the corresponding fixed affinity is 
deleted on service managers by deleting associated wild- 65 reset on the primary service manager and another replication 
cards. In some embodiments, however, this is not desired as packet is sent to the backup service manager so that the 
it may be useful to have a service manager continue to corresponding expiration period is reset on the backup 
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service manager. The backup service manager deletes 
expired fixed affinities just as the primary service manager 
deletes expired fixed affinities so that the state of fixed 
affinities on the backup service manager mirrors the state of 
fixed affinities on the primary service manager. The primary 5 
service manager is not required to notify the backup service 
manager when fixed affinities are deleted. 

It should be noted that when the backup service manager 
becomes active, the backup service manager may send a 
message to some or all of the forwarding agents deleting any to 
wildcard or fixed affinities still remaining on the forwarding 
agents that correspond to the failed primary service manger. 
The fact that the backup service manager has fixed affinities 
that correspond to each of the fixed affinities sent by the 
primary service manager enables the newly active backup 15 
service manager to send fixed affinities with immediate 
expiration times that correspond to all of the fixed affinities 
stored on forwarding agents that were previously sent by the 
failed primary service manager. 

Although the foregoing invention has been described in 20 
some detail for purposes of clarity of understanding, it will 
be apparent that certain changes and modifications may be 
practiced within the scope of the appended claims. It should 
be noted that there are many alternative ways of implement- 
ing both the process and apparatus of the present invention. 25 
Accordingly, the present embodiments are to be considered 
as illustrative and not restrictive, and the invention is not to 
be limited to the details given herein, but may be modified 
within the scope and equivalents of the appended claims. 

What is claimed is: 30 

1. A fault tolerant method of providing a network service, 
comprising: 

receiving a packet corresponding to a flow from a for- 
warding agent at a primary service manager; 

determining at the primary service manager instructions 35 
for handling packets corresponding to the flow; 

sending the instructions to the forwarding agent; 

storing the instructions for handling packets correspond- 
ing to the flow at the primary service manager; ^ 

sending a replication packet to a backup service manager, 
the replication packet including the instructions for 
handling packets corresponding to the flow; and 

deleting the instructions for handling packets correspond- 
ing to the flow from the primary service manager upon 45 
the expiration of a primary service manager instruction 
maintenance lime interval. 

2. A fault tolerant method of providing a network service 
as recited in claim 1 further including storing the instruc- 
tions for handling packets corresponding to the flow at the 50 
backup service manager. 

3. A fault tolerant method of providing a network service 
as recited in claim 1 further including deleting the instruc- 
tions for handling packets corresponding to the flow from 
the backup service manager upon the expiration of a backup 5s 
service manager instruction maintenance time interval. 

4. A fault tolerant method of providing a network service 
as recited in claim 3 wherein the backup service manager 
instruction maintenance time interval and the primary ser- 
vice manager instruction maintenance time interval are 60 
substantially the same. 

5. A fault tolerant method of providing a network service 
as recited in claim 1 further including: 

setting a primary service manager expiration time for the 
instructions for handling packets corresponding to the 65 
flow to expire on the primary service manager, the 
primary service manager expiration time being a pri- 



mary service manager instruction deletion time interval 
after the time that the instructions for handling packets 
corresponding to the flow were stored on the primary 
service manager, and 
setting a backup service manager expiration time for the 
instructions for handling packets corresponding to the 
flow to expire on the backup service manager, the 
backup service manager expiration time being a backup 
service manager instruction deletion time interval after 
the time that the instructions for handling packets 
corresponding to the flow were stoied on the backup 
service manager. 

6. A fault tolerant method of providing a network service 
as recited in claim 5 further including: 

receiving a subsequent packet corresponding to the flow 
from the forwarding agent; 

resetting the primary service manager instruction expira- 
tion time to the primary service manager instruction 
deletion time interval after the time that the subsequent 
packet was received; and 

sending a second replication packet to the backup service 
manager. 

7. A fault tolerant method of providing a network service 
as recited in claim 6 further including receiving the second 
replication packet at the backup service manager and reset- 
ting the backup service manager instruction expiration time 
to the backup service manager instruction deletion time 
interval after the time that the second replication packet was 
received. 

8. A fault tolerant method of providing a network service 
as recited in claim 1 including: 

determining at the backup service manager that the pri- 
mary service manager has failed; 

receiving at the backup service manager a subsequent 
packet from the forwarding agent corresponding to the 
flow; 

matching the subsequent packet to the instructions stored 

on the backup service manager for handling packets 

corresponding to the flow; and 
sending the instructions stored on the backup service 

manager for handling packets corresponding to the flow 

to the forwarding agent. 

9. A fault tolerant method of providing a network service 
as recited in claim 8 further including resetting a backup 
service manager instruction expiration time to a backup 
service manager instruction deletion time interval after the 
time that the subsequent packet was received. 

10. A fault tolerant method of providing a network service 
as recited in claim 8 further including sending canceling 
instructions from the backup service manager to the for- 
warding agent upon determining that the primary service 
manager has failed, the canceling instructions causing 
instructions from the primary service manager that are 
stored on the forwarding agent to expire. 

11. A primary service manager for providing a network 
service in a fault tolerant manner, comprising: 

a processor configured to determine instructions for han- 
dling packets corresponding to a flow; 

a forwarding agent interface configured to send the 
instructions for handling packets to a forwarding agent; 

a memory configured to store the instructions for handling 
packets corresponding to the flow; and 

a backup service manager interface configured to send a 
replication packet to a backup service manager wherein 
the replication packet includes instructions for handling 
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packets corresponding to the flow, and wherein the expire on the backup service manager, the backup 

primary service manager is further configured to delete service manager expiration time being a backup service 

the instructions for handling packets corresponding to manager instruction deletion time interval after the time 

the flow upon the expiration of a primary service that the instructions for handling packets corresponding 

manager instruction maintenance time interval. 5 to the flow were stored on the backup service manager. 

12. A backup service manager for providing a network 17. A fault tolerant distributed system as recited in claim 
service in a fault tolerant manner, comprising: 16 wherein the primary service manager is further config- 

a primary service manager interface configured to receive ured t0 receive a subsequent packet corresponding to the 

the instructions for handling packets corresponding to flow from tbe forwarding agent, reset the primary service 

a flow and 10 manager instruction expiration time to the primary service 

„ c . . ... ,, . , , , „. manager instruction deletion time interval after the time that 

a memory configured to store the instructions for handling iU . , , . 

, , j. , t . a , . . the subsequent packet was received and send a second 

packets corresponding to the flow, where in the primary t - i \ „ tL L , 

• „„„„„„ • fi, rf t,„ . ,„ i T ,„ : replication packet to the backup service manager. 

service manager is turther configured to delete the \ a , . K . , t t ., A *\ ^ ° , . . . 

eZ. u«„^u „i r „,„ ° „ ,„ 18. A fault tolerant distributed system as recited in claim 

instructions tor handling packets corresponding to the n» i_ ■ „. . • • ^ j 

r. „„ lc • , • JL„„„„ is 17 wherein the backup service manager is further configured 

now upon the expiration or a primary service manager 4 . , j ■ r , . . , 

instruction maintenance time interval. to receive tbe ^ "Phcalion packet and reset the backup 

13. A fault tolerant distributed system for providing a mana g er Action expuahon time to the backup 

' r & service manager uistruction deletion time interval after the 

network service including: . , , ,• , . 

time that the second replication packet was received. 

a forwarding agent configured to send a packet corre- 2Q 19 A faul| tolerant distributed system as reciled m claim 

spending to a flow to a primary service manager, 13 wherein , he backup XTV ice manager is father configured 

a primary service manager configured to determine to determine that the primary service manager has failed, to 

instructions for handling packets corresponding to the receive a subsequent packet from the forwarding agent 

flow, to send the instructions for handling packets to the corresponding to the flow, to match the subsequent packet to 

forwarding agent, to store the instructions for handling 2 s the instructions stored on the backup service manager for 

packets corresponding to the flow and to send a repli- handling packets corresponding to the flow; and to send the 

cation packet to a backup service manager, the repli- instructions stored on the backup service manager for han- 

cation packet including the instructions for handling dling packets corresponding to the flow to the forwarding 

packets corresponding to the flow; and agent. 

a backup service manager configured to receive the 30 20. A fault tolerant distributed system as recited in claim 

instructions for handling packets corresponding to the 19 wherein the backup service manager is further configured 

flow and to store the instructions for handling packets upon determining that the primary service manager has 

corresponding to the flow, wherein the primary service failed to send instructions to the forwarding agent that cause 

manager is further configured to delete the instructions instructions from the primary service manager that are 

for handling packets corresponding to the flow upon the 35 stored on the forwarding agent to expire, 

expiration of a primary service manager instruction 21. A fault tolerant distributed system as recited in claim 

maintenance time interval. 19 wherein the backup service manager is further configured 

14. A fault tolerant distributed system as recited in claim to reset the backup service manager instruction expiration 

13 wherein the backup service manager is further configured time to a backup service manager instruction deletion time 
to delete the instructions for handling packets corresponding 40 interval after the time that the subsequent packet was 
to the flow upon tbe expiration of a backup service manager received. 

instruction maintenance time interval. 22. A computer program product for providing a network 

15. A fault tolerant distributed system as recited in claim service in a fault tolerant manner embodied in a computer 

14 wherein the backup service manager instruction mainte- readable medium comprising computer instructions for: 
nance time interval and the primary service manager ins true- 45 receiving a packet corresponding to a flow from a for- 
tion maintenance time interval arc substantially the same. warding agent at a primary service manager; 

16. A fault tolerant distributed system as recited in claim determining at the primary service manager instructions 
13 wherein: for handling packets corresponding to the flow; 

the primary service manager is further configured to set a sending the instructions to the forwarding agent; 

primary service manager expiration time for the 50 storing tbe instructions for handling packets correspond- 

instructions for handling packets corresponding to the ing to the flow at the primary service manager; and 

flow to expire on the primary service manager, the sending a replication packet to a backup service manager, 

primary service manager expiration lime being a pri- the replication packet including the instructions for 

maty service manager instruction deletion time interval handling packets corresponding to the flow, wherein 

after the time that the instructions for handling packets 55 the primary service manager is further configured to 

corresponding to the flow were stored on the primary delete the instructions for handling packets correspond- 

service manager; and m g l0 the flow upon the expiration of a primary service 

the backup service manager is further configured to set a manager instruction maintenance time interval, 
backup service manager expiration time for the instruc- 
tions for handling packets corresponding to the flow to ***** 
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